Add multicam image registration backend by romleiaj · Pull Request #1750 · Kitware/dive

romleiaj · 2026-07-10T16:51:58Z

No description provided.

homography.ts: 3x3 matrix primitives (multiply, invert, apply, linear and DLT solvers) plus warp-grid subdivision helpers. transform.ts: keypointgui-style transform models (translation / rigid / similarity / affine / homography) that all estimate to a plain Matrix3, with Similarity as the single shared default. CameraCalibrationStore: the persistence core for camera-rig calibration -- per-pair homographies, correspondences, transform types, and producer provenance, with hydrate() from dataset meta, dirty tracking, and the portable calibration.json round trip (toCalibrationJson / loadCalibrationText). Interactive creation (point picking, fitting, overlay preview) is not part of this branch; it layers on top in the Manual Alignment GUI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Persist cameraHomographies / cameraCorrespondences / cameraTransformTypes / cameraCalibrationSource in dataset meta on both platforms (server allowlist + desktop meta merge), and load calibrations produced externally (e.g. by kamera) at import time: the multicam import dialog gains a per-camera transform-file picker, the desktop backend parses DIVE calibration .json pairs (fromCalibrationPairs / toCalibrationPairs) and auto-discovers a calibration.json in the import parent folder, and saveMetadata writes the portable calibration.json alongside the project so the calibration travels with the dataset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

When every camera resolves a transform into the reference camera's space (first camera in display order, composed through the calibration pair graph), an Align button warps each pane's imagery and annotations into that shared space at draw time -- stored geometry always stays native per camera. Includes: AlignedViewStore + resolution wiring in the Viewer; AlignedImageLayer (sub-quad warped imagery with seam overlap); display-transform routing through every annotation layer; transform-aware linked pan/zoom (useAlignedNavigation/useLinkedViewers, with the raw screen-delta camera sync retired once transforms exist); drawing/editing on any camera with edits mapped back through the inverse; and a continuous cross-camera mirror that re-projects every geometry edit onto all other calibrated cameras. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Importing single-camera annotations into a calibrated multicam dataset now offers an "Import to all cameras" checkbox: after the import lands on the active camera, every track is copied onto each other camera through the camera-to-camera homographies (bounds, rotated bounds, and GeoJSON geometry), then saved. Unlike the Align View continuous mirror this is a data import, so confidence pairs and track/feature attributes are copied too. Gated on the whole rig resolving to the reference (alignedView.available), independent of the display toggle. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The dataset's saved calibration moves from a single all-pairs calibration.json to one calibration_<camera>.json per non-reference camera (a pair not touching the reference files under its right camera), matching the per-camera files users already handle from kamera. File names are discovery/provenance only: the pair's left/right names inside each file body stay authoritative, so a misnamed or copied file can never rebind a transform to the wrong camera. The legacy single file is still read and is migrated (rewritten per-camera, then removed) on the next save; stale per-camera files are cleaned up so the on-disk set always mirrors the saved calibration. Per-file producer stamps are compared when the set is merged (project load and import seeding): agreement keeps the stamp, disagreement becomes a { mixed: true, files: {...} } composite that the Align View tooltip surfaces as a mixed-generation warning instead of composing silently. A save never writes the composite into per-camera files -- that would read back as a unanimous rig. Parent-folder transform discovery now returns every self-identified calibration file (calibration.json first, then per-camera files), and the multicam import dialog auto-attaches calibration_<camera>.json to its matching camera slot. Files written by DIVE now also self-identify with type: dive-camera-calibration so discovery recognizes them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The per-camera calibration_<camera>.json files are now the only on-disk form: the single all-pairs calibration.json is no longer read, migrated, or given priority during import discovery. The Export menu gains explicit calibration export -- one file per calibrated camera, or all of them zipped -- built from the same per-camera grouping the save path uses, and the multicam Everything zip now carries the files too. Exports resolve the calibration the same way loading does (standalone files first, the import-time meta seed otherwise) and never stamp a file with a mixed composite source. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Calibration export is now strictly per-camera -- one calibration_<camera>.json at a time, no all-cameras zip -- and works on both platforms: desktop writes through the backend as before, web builds the file client-side from the dataset meta and downloads it. The Import menu gains a matching per-dataset calibration import (same workflow as annotation import): pairs the file names replace the current ones, other cameras' pairs are kept, so per-camera files can be imported one at a time to assemble a rig; the viewer store rehydrates immediately so the Align View picks up the new transforms without a reload. Grouping and merge semantics live in a new shared vue-media-annotator module used by the desktop backend, both Export menus, and both import paths, so listed cameras, written files, and merge behavior cannot drift between platforms. Web needs no server changes: the calibration meta fields were already allowlisted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The Import and Export buttons now advertise "Supplementary Data" in their hover tooltips instead of "Annotation Data", since both menus carry calibrations as well. The Import Calibration blurb shrinks to one line. The warp-on-import checkbox moves out of the active-camera alert to sit beside Overwrite as "Warp to All", always visible on multicam datasets and greyed out until the whole rig resolves, with an "N/M cameras calibrated" hint. The viewer publishes that count into AlignedViewStore from the same computation as the Align View tooltip, so the two indicators can never disagree. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The generic Import Calibration button becomes one button per non-reference camera pair ("Import eo -> ir"), each scoped to its camera: only the file's pairs naming that camera are taken, they replace that camera's current pairs, and other cameras' pairs are kept -- so per-camera files assemble a rig one at a time and a multi-pair file pointed at one camera doesn't drag the rest along. Scoping to a camera the file doesn't name refuses with a clear error. Buttons are green when the camera already has a calibration and yellow when it doesn't, and importing onto an existing one asks for confirmation before replacing it, so no separate overwrite toggle is needed. The camera filtering and pair/camera summary live in the shared calibration-file module so desktop and web behave identically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ormat "Calibration" collided with the pre-existing stereo camera file (an intrinsic/extrinsic camera model for the measurement pipeline), and in computer vision this process is image registration, so the feature says registration everywhere: "Import Registration" / "Registration: <camera>" menu sections, importCameraRegistration and exportCameraRegistration API members and IPC channels, registrationProgress ("N/M cameras registered") published on the aligned-view store, matching error messages, the internals (CameraRegistrationStore, cameraRegistrationFiles, and the load/merge/build helpers), and the wire format: the dataset meta field is cameraRegistrationSource, renamed through the server models, allowlist, and tests alongside the RegistrationSource type and stamp helpers. Files are named <camera>_to_<reference>_registration.json and self-identify as dive-camera-registration: each file states the destination of the mapping it carries (ir_to_eo_registration.json registers ir onto eo), making the reference-camera rule -- first camera in display order -- legible from the artifacts themselves. The name scheme contains neither "calibration" nor "cal", so these files can no longer be mis-suggested as stereo camera files by the stereo parent-folder discovery heuristic. The Align View keeps its name (registration is the data and process; the Align View is the display mode that renders registered imagery), and the stereo camera-file surfaces are untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The registration reference was implicitly the first camera in display order, which merely coincided with the camera users pick at import in the common EO-first case. The import dialog's picker is now labeled "Reference Camera" ("Choose which camera to use as the reference camera for image registration") and its choice -- stored as the dataset's defaultDisplay -- actually drives the reference, through a shared referenceCameraName() resolver used by the viewer's transform resolution and N/M-registered status, both Export menus, and the desktop per-camera file persistence, falling back to display order when the choice is missing. The viewer now publishes the reference on the aligned-view store even before the rig resolves, so the per-pair import buttons name it ("Import ir → eo") instead of assuming camera-map order. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

BryonLewis

These comments are for myself to go back and look more closely at. If you want to implement in Claude let me know, if not I'll work on it.

Most of the comments are structural to reduce large files and better organize this code into some folders to separate the feature as well as add some README.md files to them to give a basic outline of what the hundreds of new lines do.

BryonLewis · 2026-07-10T18:25:06Z

    const groupStyleManager = new StyleManager({ markChangesPending, vuetify });

    const cameraStore = new CameraStore({ markChangesPending });
+    const cameraRegistration = new CameraRegistrationStore();


wondering if some of this can be placed behind multicamera dataset check so it isn't for all datasets?

BryonLewis · 2026-07-10T18:45:57Z

  await fs.writeFile(absPath, serialized);
 }

+type CameraHomographies = NonNullable<DatasetMetaMutable['cameraHomographies']>;


wondering if in this file alot of the functions and sections of conditionals should be moved to another file to reduce the overall complexity of common.ts?

BryonLewis · 2026-07-10T18:55:44Z

          </v-btn>
        </template>
-        <span> Export Annotation Data </span>
+        <span> Export Supplementary Data </span>


double check this

BryonLewis · 2026-07-10T19:13:12Z

@@ -0,0 +1,105 @@
+import { Ref, watch } from 'vue';


for this an useLinkedViewer.ts lets place them inside their own subfolder and add a README.md explaining what they are doing

BryonLewis · 2026-07-10T19:16:04Z

+      const layerList = viewer.layers();
+      for (let i = 0; i < layerList.length; i += 1) {
+        const layer = layerList[i];
+        if (typeof layer.features === 'function') {
+          const features = layer.features();
+          for (let j = 0; j < features.length; j += 1) {
+            const data = typeof features[j].data === 'function' ? features[j].data() : undefined;
+            const datum = Array.isArray(data) ? data[0] : undefined;
+            if (datum && datum.image) {
+              const image = datum.image as HTMLImageElement;
+              return {
+                source: image, kind: 'image' as const, width: image.naturalWidth, height: image.naturalHeight,
+              };
+            }


check and see if there is an easier way to do this than nested for loops. if not lets move this into another file

BryonLewis · 2026-07-10T19:17:37Z

+      // Right-click means "remove last point" while creating/editing
+      // geometry; recenter everywhere else.
+      getRecenterEnabled: () => !editingModeRef.value,
+    });


can this be refactored into CameraTransformLayerManager or something else to declutter the already big LayerManager?

BryonLewis · 2026-07-10T19:23:36Z

@@ -0,0 +1,165 @@
+// eslint-disable-next-line import/no-extraneous-dependencies -- Vitest is only used in tests


alignedView, AlignedViewStore, cameraRegistrationFiles, CameraRegistrationStore, homography, transform should go into it's own subfolder with a README.md explaining what each file does

BryonLewis · 2026-07-10T19:32:15Z

+    # Per-camera-pair alignment homographies, keyed by directional "left::right".
+    # Each value holds the 3x3 AtoB / BtoA matrices.
+    cameraHomographies: Optional[Dict[str, Dict[str, List[List[float]]]]]
+    # The picked point correspondences behind those homographies, keyed the same
+    # way. Each entry is a list of {id, a: [x, y], b: [x, y]} pairs.
+    cameraCorrespondences: Optional[Dict[str, List[Dict[str, Any]]]]
+    # The fit model used to compute each pair's homography (translation / rigid /
+    # similarity / affine / homography), keyed the same way. Missing entries
+    # default to 'similarity' client-side.
+    cameraTransformTypes: Optional[Dict[str, str]]
+    # Free-form producer provenance for the camera calibration (e.g. an external
+    # model step's version / swathe / generation time). Never interpreted by
+    # DIVE; preserved verbatim so refined calibrations can be traced back to the
+    # model version they were made against.
+    cameraRegistrationSource: Optional[Dict[str, Any]]


check and see if any of these types can be more tightly defined

romleiaj and others added 9 commits July 9, 2026 16:16

romleiaj marked this pull request as draft July 10, 2026 16:52

romleiaj mentioned this pull request Jul 10, 2026

Add multicam calibration #1748

Closed

romleiaj and others added 2 commits July 10, 2026 13:00

romleiaj force-pushed the sealtk/image-registration branch from 30ebbee to a6d545d Compare July 10, 2026 18:38

BryonLewis requested changes Jul 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add multicam image registration backend#1750

Add multicam image registration backend#1750
romleiaj wants to merge 11 commits into
dev/sealtkfrom
sealtk/image-registration

romleiaj commented Jul 10, 2026

Uh oh!

BryonLewis left a comment

Uh oh!

BryonLewis Jul 10, 2026

Uh oh!

BryonLewis Jul 10, 2026

Uh oh!

BryonLewis Jul 10, 2026

Uh oh!

BryonLewis Jul 10, 2026

Uh oh!

BryonLewis Jul 10, 2026

Uh oh!

BryonLewis Jul 10, 2026

Uh oh!

BryonLewis Jul 10, 2026

Uh oh!

BryonLewis Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,165 @@
		// eslint-disable-next-line import/no-extraneous-dependencies -- Vitest is only used in tests

Uh oh!

Conversation

romleiaj commented Jul 10, 2026

Uh oh!

BryonLewis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants